Job runs wait for completion - #6091

Deploying a job_run triggered the run and moved on, so a resource referencing
the run's outcome saw whatever state the run happened to be in. The resource now
implements the framework's WaitAfterCreate hook, which blocks until the run is
terminal and republishes the settled remote state; only SUCCESS lets the deploy
continue.

While it waits, the deploy reports the run page URL and each state change, the
way `bundle run` does, since a run can take hours. A run that does not succeed
fails the deploy with the failed task, the message that task reported, and a link
to the run page. Bounded by 24h, matching `bundle run`.

The framework saves the run id before the wait, so a run that fails stays
recorded and an unchanged config plans no second run; failed_run covers that.

run_page_url is now normalized to the path form that also resolves for
non-admins.

invariant/configs/job_run.yml.tmpl loses its notebook task, which deploying it
would now actually run, and is excluded from cloud runs: a real workspace reports
a condition-task-only run as SKIPPED, and a task that does succeed would add a
multi-minute cluster run to every variant of a suite that asserts plan and state
invariants. Still covered locally.

The wait reimplemented what `bundle run`'s monitor already does: report the run
page URL once, then each state change. Both now go through
progress.JobStateTracker, which decides what a poll is worth reporting; the two
callers keep their own sinks, since a concurrent deploy reports plain prefixed
lines where `bundle run` reports progress events.

The failure the deploy reports no longer repeats the run id that the framework's
wrapper already carries; what the wait adds is the link to a run that outlives it.

Excluding job_run.yml.tmpl from cloud left job_runs with no cloud coverage at
all: every test under resources/job_runs inherits Cloud=false. The behaviour
this milestone adds is the one that depends most on the real Jobs API, so
wait_cloud triggers a run for real and reads the outcome back out of the
downstream job, the way the vector search exclusion points at a dedicated test.

Serverless keeps the run to about a minute, and the deploy's progress stream
stays out of the golden: a real run reports an unpredictable number of
intermediate states.

The ignore_remote_changes for job_parameters assumes GetRun reports every
parameter the job defines, not just the ones the run overrode. Assert that on the
run wait_cloud already triggers by overriding one of two parameters and planning
after the deploy.

Drop the comments that only restate the code, the duplicated note about the
framework prefixing the run id, and the filler in the ones that carry a reason.

Reporting every state change made a deploy's output depend on how many states the
run passed through, which varies with how long its compute takes to start. That
cost the cloud test its coverage: it had to grep the deploy log instead of
comparing it. The full state history is still in the log.

Also record that a failed run is not run again in the changelog entry.

The message the deploy names a failed task with came from an error the test server
writes itself, so nothing checked that a real workspace reports one at all. Assert
it does, and that we are not falling back to the states the run reports.

Also fix jobRunServer's doc, which said it returns a server when it returns a
client.

Interrupting a deploy mid-wait leaves the run going, and jobs/runs/delete rejects
an active run, so destroy failed and the bundle could not be torn down without
cancelling the run by hand. Delete now cancels it first, and waits for the
cancellation to settle since the API cancels asynchronously.

The interrupt itself was reported as a timeout, blaming the 24h bound for
something the user did. It now says it was interrupted, and still links the run,
whose page URL is pinned to the first poll that reported one.

The run left going is what the next deploy reads, and it triggers no second run,
so a reference to the outcome resolves to an empty string. Recorded in a test
rather than fixed here: stopping a run the user did not ask to stop is a
departure from `bundle run`, which leaves interrupted runs alive.

A real workspace reports a run whose task failed as INTERNAL_ERROR in the
deprecated life_cycle_state, though status.state is TERMINATED with termination
code RUN_EXECUTION_ERROR. The SDK waiter halts on INTERNAL_ERROR with an error of
its own, so the deploy blamed the run for an internal failure instead of naming
the task that failed and the message it reported. The wait now polls for any
state runIsTerminal accepts, the definition the delete path already used, and
leaves the verdict to the run's result.

The Jobs API retries a task that failed and reports it once per attempt, so the
same task was named twice over. Only its last attempt is reported now.

The fake workspace rolls a failed task up to TERMINATED FAILED, so failed_cloud
was the only test that saw either of these; both are now covered by unit tests.

Interrupting a deploy mid-wait leaves the run going and its id recorded, so the
next deploy plans no change for it and serves references to it from the remote
state cache. An unfinished run reports an empty result_state, and that was
substituted into whatever referenced it, configuring a downstream resource with
the outcome of a run that had not reached one.

Resources now declare when their outputs are not final yet, through an optional
CheckSettled that mirrors IsGone, and reference resolution fails with what the
resource reports instead of handing out the zero value. job_runs answers with
runIsTerminal, the definition the wait loop already polls for, so the two cannot
drift apart, and names the run and links its page.

The check sits at reference resolution rather than in DoRead or the planner, so
it only fires for a bundle that actually reads the outcome. A deploy that merely
carries the run keeps working, as do plan, summary and destroy, while the run
finishes on its own; cancelling it would be the departure from `bundle run` that
this branch already declined to make.

wait_cloud inherited Local=true, so both tests deployed the same job_run
against the test server and differed only in the assertions that followed. The
merged test keeps wait_output's coverage that the resolved result_state tag is
not perpetual drift, and reads the tag back with `jobs get` rather than from
recorded requests, so that assertion holds against a real workspace as well.

Recording requests stays off: the wait polls GetRun until the run is terminal,
so the recorded requests depend on how long the run takes.

The _cloud suffix goes with it, since the test was never cloud-only.

Comparing WaitAfterCreate's return against a fresh read, and rebasing the field
checks onto it, applied to every resource, but job_runs is the only one whose
remote state moves while the wait blocks: the testserver settles the others on
create, so both forms assert the same thing for them.

Keep the original comparison for those and take the new path only for job_runs,
so the change stays where it is needed. A resource whose handler later models a
create-then-settle transition will fail the plain comparison and needs adding to
the exception.

Comparing WaitAfterCreate's remote against a fresh DoRead held by construction:
both build the value with makeJobRunRemote from a GetRun response, and the run
is terminal by then, so the two cannot differ. It also did not check the thing
worth checking -- an early return from the wait would leave both sides equally
non-terminal -- and the drift a lossy recorded state would cause is already
covered by resources/job_runs/{basic,wait}.

What remains is not an assertion: result_state fills in only once the run
settles, so the field checks at the end of testCRUD have to read the settled
state rather than the one from just after create.

The wait borrowed `bundle run`'s progress event to report a run, which put a
timestamp in the middle of a deploy's output, made dresources depend on the
`bundle run` progress package, and repeated on failure the state message the
error already carries. A deploy now prints the run page and, once the run
succeeds, its outcome; every state change still goes to the log. bundle/run is
untouched again, so the shared JobStateTracker is no longer needed.

Review cleanups along with it:

- ignore_remote_changes says why each field is ignored rather than calling all
  of them immutable: job_id is, the parameter fields come back resolved against
  the job's defaults, and the rest are never returned. The reason is what
  `bundle plan` shows the user.
- Restore the note that `field: ""` matches nothing, which still applies to the
  root recreate_on_changes rule.
- Drop TestLookupReferencePreDeploy_JobRunReferencesAreRaw, left over from the
  CheckSettled iteration: it asserts framework behaviour this branch does not
  change.
- testCRUD reads the resource again after WaitAfterCreate instead of special
  casing job_runs, the way the DoUpdate block below it already does.
- Drop the RecordRequests failed_run already inherits, and comment the read_id.py
  call that exists only to register a replacement.

…d to job_runs

The previous commit made every resource compare WaitAfterCreate's return against
a fresh read and rebased the field checks onto it, but job_runs is the only one
whose remote state moves while the wait blocks: the testserver settles the others
on create, so the original assertion already says the same thing for them.

Restore that assertion and take the settled state only for job_runs, whose
result_state fills in once the run finishes and which the field checks below
therefore need.

A run has a handful of tasks, so scanning the slice already built is cheaper
than a map plus a second slice to keep the order it was filled in.

State is deprecated in favour of Status. Both are set today, so taskFailed
took the simpler one and taskError read its message; the day a workspace
stops setting State, every failed task would go unreported and the run's own
generic message would be all that is left. Fall back to Status instead.

newTaskFailure read the exception off the last line of the task's output, so
a task that exited non-zero silently was reported with an empty message.

An interrupted wait leaves the run going, and DoDelete cancels it before
deleting it, since jobs/runs/delete rejects an active run. The fake had no
handler for that call, so the path had no local coverage.

The condition task needed no compute, but a real workspace reports a run with
no material task to execute as SKIPPED, which fails the wait. A serverless
Python task keeps the run to about a minute.

Several comments on this branch explained the option that was not taken (the
SDK waiter, a condition task, reporting the whole output as the error). Drop
the alternatives and state the behaviour the code has.

A deploy interrupted while waiting leaves a run going, and its empty
result_state read as drift, so the next deploy cancelled a run that could
still succeed and triggered another. OverrideChangeDesc now downgrades that
drift to an update while the run has not reached a terminal lifecycle state,
and WaitAfterUpdate resumes the wait on it.

DoUpdate existed only because DeploymentUnit.Update refuses to run without
it, so the wait it enables lived in a second hook. One method that waits
says the same thing as an empty one plus WaitAfterUpdate.

OverrideChangeDesc is a pure function over (path, change, remote), but the
interrupted_run acceptance test was its only coverage, which is why proving
a 9-line rule needs fault injection and a saved plan.

The in-progress rule is already covered through the planner in
bundle_plan_test.go, which reaches more lifecycle states than the direct
calls added on top of it.

The old comment read as if the run itself failed, which is confusing next to
the failed_run test.

The SDK's WaitGetRunJobTerminatedOrSkipped drops the run on INTERNAL_ERROR,
which is the state a run whose task failed lands in, so runFailedError would
have nothing left to name the task from. Also fold the empty-result_state
fallback into cmp.Or and drop a single-use flag in the CRUD harness.

Local is no longer a test.toml key; keeping it fails config loading.

Keep result_state as a planning sentinel so failed runs still recreate, but
do not write it to resources.json. Refresh remote after WaitAfterCreate in
testCRUD, and dump the persisted job_run state in basic/interrupted_run.

resources/job_runs/failed_run failed on every cloud. A real workspace sends
"user_cancelled_or_timedout": false, which the SDK records in ForceSendFields,
so copying GetRun's state into the remote serialized that field against a
workspace and not against the testserver. Build the state from the three
fields the resource reads instead.

run_page_url needed a replacement too: failed_run records requests, so on
cloud the CLI is configured with the proxy's host and [DATABRICKS_URL] no
longer matches the workspace host the Jobs API reports.